{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2-mnist-with-keras-eager-and-tf-data.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "2-B5UZxUchfd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# MNIST with tf.keras, tf.data, and eager execution.\n",
        "\n",
        "In this lab, you'll learn how create a `tf.data` [Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) for the MNIST dataset. The tf.data API provides tools for working with data (including common functionaltity, like shuffling, batching, etc), as well as high performance utilities (like parallel reads, and pre-fecthing to GPUs) - should you need these down the road. Here, we'll wrap MNIST with tf.data, just to introduce the API. Later, you'll learn how to use tf.data on your own data, say, to read a directory of images off disk.\n",
        "\n",
        "To learn more about tf.data, you can check out [this](https://www.youtube.com/watch?v=EHHdyM3NNiA&index=26&list=PLQY2H8rRoyvxjVx3zfw4vA4cvlKogyLNN) talk from the [2018 TensorFlow Developer Summit](https://www.tensorflow.org/dev-summit/).\n",
        "\n",
        "The rest of the code similar to the previous notebook, with three changes. \n",
        "\n",
        "1) Instead of using `model.fit`, we use `model.train_on_batch` as we iterate over our dataset. \n",
        "2) We will create a TensorFlow optimizer to pass to the Keras model.\n",
        "3) We will enable eager execution.\n",
        "\n",
        "### Eager execution\n",
        "Eager execution is a mode for running TensorFlow that works just like regular Python. To learn more about eager, check out [this talk](https://www.youtube.com/watch?v=T8AW0fKP0Hs&list=PLQY2H8rRoyvxjVx3zfw4vA4cvlKogyLNN&index=8)."
      ]
    },
    {
      "metadata": {
        "id": "902Rjd5DZroO",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install -q -U tensorflow==1.8.0\n",
        "import tensorflow as tf\n",
        "\n",
        "# Enable eager execution\n",
        "tf.enable_eager_execution()\n",
        "\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "evWlYUkYefG8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JA3braIeeggc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "TRAINING_SIZE = len(train_images)\n",
        "TEST_SIZE = len(test_images)\n",
        "\n",
        "# Reshape from (N, 28, 28) to (N, 784)\n",
        "train_images = np.reshape(train_images, (TRAINING_SIZE, 784))\n",
        "test_images = np.reshape(test_images, (TEST_SIZE, 784))\n",
        "\n",
        "# Convert the array to float32 as opposed to uint8\n",
        "train_images = train_images.astype(np.float32)\n",
        "test_images = test_images.astype(np.float32)\n",
        "\n",
        "# Convert the pixel values from integers between 0 and 255 to floats between 0 and 1\n",
        "train_images /= 255\n",
        "test_images /=  255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wbwRqi0BeicT",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "NUM_DIGITS = 10\n",
        "\n",
        "print(\"Before\", train_labels[0]) # The format of the labels before conversion\n",
        "\n",
        "train_labels  = tf.keras.utils.to_categorical(train_labels, NUM_DIGITS)\n",
        "\n",
        "print(\"After\", train_labels[0]) # The format of the labels after conversion\n",
        "\n",
        "test_labels = tf.keras.utils.to_categorical(test_labels, NUM_DIGITS)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XFtvnaI5jU_5",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Cast the labels to floats, needed later\n",
        "train_labels = train_labels.astype(np.float32)\n",
        "test_labels = test_labels.astype(np.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jjNr3gr3ejh2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "model = tf.keras.Sequential()\n",
        "model.add(tf.keras.layers.Dense(512, activation=tf.nn.relu, input_shape=(784,)))\n",
        "model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))\n",
        "\n",
        "\n",
        "# Create a TensorFlow optimizer, rather than using the Keras version\n",
        "# This is currently necessary when working in eager mode\n",
        "optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001)\n",
        "\n",
        "# We will now compile and print out a summary of our model\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer=optimizer,\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zUZPvDQYe5Xu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 1) Create a tf.data Dataset\n",
        "\n",
        "Here, we'll use the `tf.data.Dataset` [API](https://www.tensorflow.org/api_docs/python/tf/data) to convert the Numpy arrays into a TensorFlow dataset.\n",
        "\n",
        "Next, we will create a simple for loop that will serve as our introduction into creating custom training loops. Although this essentially does the same thing as `model.fit` it allows us to get creative and customize the overall training process (should you like to, if you venture into research) and collect different metrics throughout the process."
      ]
    },
    {
      "metadata": {
        "id": "sdBd2pd_fdue",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "BATCH_SIZE=128\n",
        "\n",
        "# Because tf.data may work with potentially **large** collections of data\n",
        "# we do not shuffle the entire dataset by default\n",
        "# Instead, we maintain a buffer of SHUFFLE_SIZE elements\n",
        "# and sample from there.\n",
        "SHUFFLE_SIZE = 10000 \n",
        "\n",
        "# Create the dataset\n",
        "dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))\n",
        "dataset = dataset.shuffle(SHUFFLE_SIZE)\n",
        "dataset = dataset.batch(BATCH_SIZE)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ksAR-C6xgUu4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 2) Iterate over the dataset\n",
        "Here, we'll iterate over the dataset, and train our model using `model.train_on_batch`. To learn more about the elements returned from the dataset, you can print them out and try the `.numpy()` method.\n"
      ]
    },
    {
      "metadata": {
        "id": "kNgnUKPvgSCz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "EPOCHS=5\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  for images, labels in dataset:\n",
        "    train_loss, train_accuracy = model.train_on_batch(images, labels)\n",
        "  \n",
        "  # Here you can gather any metrics or adjust your training parameters\n",
        "  print('Epoch #%d\\t Loss: %.6f\\tAccuracy: %.6f' % (epoch + 1, train_loss, train_accuracy))\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tg5U3Iqkgo3J",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "loss, accuracy = model.evaluate(test_images, test_labels)\n",
        "print('Test accuracy: %.2f' % (accuracy))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Fu0eSNa9kEFZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Congratulations\n",
        "You have trained a model on MNIST using Keras, eager execution, and tf.data."
      ]
    }
  ]
}